A Framework for Real-Time Face and Facial Feature Tracking using Optical Flow Pre-estimation and Template Tracking

This thesis presents a framework for tracking head movements and capturing the movements of the mouth and both the eyebrows in real-time. We present a head tracker which is a combination of a optical flow and a template based tracker. The estimation of the optical flow head tracker is used as starting point for the template tracker which fine-tunes the head estimation. This approach together with re-updating the optical flow points prevents the head tracker from drifting. This combination together with our switching scheme, makes our tracker very robust against fast movement and motionblur. We also propose a way to reduce the influence of partial occlusion of the head. In both the optical flow and the template based tracker we identify and exclude occluded points. When the position and orientation of the head is known, we use this together with the 3D model to find the mouth and eyebrow movements. We use the head estimation to create a rectified image (RI) patch of the mouth and eyebrow regions. The mouth tracking is done by finding the right mouth model deformations so that the initial (closed) mouth is reconstructed. Eyebrow tracking is done by minimizing an error function which depends on template similarity and an eyebrow shape constraint. To evaluate our framework and its different trackers, we have conducted experiments using the well known Boston Head Tracking database [1] and our own dataset. The results show that our tracker can successfully track the head of different people. Also, the combination of the optical flow and the template based trackers reduces the number of head losses significantly when having fast movement or motion-blur. Excluding occluded points from the tracking process makes the tracker less sensitive to occlusion and makes the “head lost” detection more robust. The results of the mouth and eyebrow trackers show that basic expressions can be captured and that they can recover very well after erroneous tracking. Furthermore, we have evaluated the computation time of each component of the framework and measured an average total computation time of 35 ms, that is 29 fps.